So I already describe this in another post and there are various guides on the intarwebz: 1, 2.
But I wanted to point out one thing. For ubuntu, the pam_access line wasn’t working in pam.d/common-auth, where I usually put it, nor was it working in pam.d/login.
But, it worked in /etc/pam.d/common-account !!!
Just throw it below the pam_deny.so,
account [success=1 newauthtokreqd=done default=ignore] pamunix.so account requisite pamdeny.so account required pamaccess.so account required pampermit.so
^is kinda what mine looks like