I’m curious too
It wouldn't be very hard to arrive on a 90 visa free entry stamp and volunteer on a farm or something like that. They would provide housing and food, but no visa or money.
Getting a visa is the tough part. You usually have to have some special skill - that’s why many teach English. Otherwise employers will just take from the local population who can speak Japanese and have a passport.
If you are a student in Japan, then getting work at a convenience store seems to be fairly straightforward. The requirement seems to be a reasonable level of Japanese competence. They appear to receive a fair bit of training. The Japanese government is also introducing new ways for foreigners to work in Japan in a variety of industries. I don’t have much information about this but a search on job websites might give you more information.
As for higher level jobs, well for one thing you will often need a pretty good level of Japanese ability. At least Level Two of the Japanese Proficiency Test is a minimum for many banking and finance jobs. However, this is not always the case. I know a few people that work in the recruiting and programming fields. They have reasonable, but by no means exceptional Japanese skills. It depends on whether you will need to use Japanese on a day to day basis. My friends that don't teach english work primarily in an English-speaking environment or by themselves on desk jobs.
How easy is it to land a non-teaching job in Japan? I think it depends on the field and how much demand there is for your skills.