Yes, you can leave lists that were shared with you. You cannot leave a list if you are the list owner!
Find the list your want to leave and tap on the options button
Select Leave List.
Once you leave a List, that list will no longer appear under Shared with me lists