You really shouldn't, they need a full nights sleep just like we do. If you're very careful you can take them out and have some sleeping cuddle time, but for the most part handling time should be confined to daytime hours.
BTW, they need about 14 hours of daylight... Are your lights coming on at 6:30 am?