So I already describe this in another post and there are various guides on the intarwebz: 1, 2.
But I wanted to point out one thing. For ubuntu, the pam_access line wasn’t working in pam.d/common-auth, where I usually put it, nor was it working in pam.d/login.
But, it worked in /etc/pam.d/common-account !!!
Just throw it below the pam_deny.so,
account [success=1 new_authtok_reqd=done default=ignore] pam_unix.so account requisite pam_deny.so account required pam_access.so account required pam_permit.so
^is kinda what mine looks like