Use the Keyword Planner to flag any terms on your list that have way too little (or way too much) search volume, and don't help you maintain a healthy mix like we talked about above. But before you delete anything, check out their trend history and projections in Google Trends. You can see whether, say, some low-volume terms might actually be something you should invest in now -- and reap the benefits for later.
Additionally, there are many situations where PPC (a component of SEM) makes more sense than SEO. For example, if you are first launching a site and you want immediate visibility, it is a good idea to create a PPC campaign because it takes less time than SEO, but it would be unwise to strictly work with PPC and not even touch search engine optimization.