In my days of clinical medicine, there were times when I would ask a patient what allergies does he or she have, and the patient would tell me medications that produce side effects. It's enough to let me know that people may think they know what an allergy is but they might not. More precisely, they know what an allergy is but they think it also includes things that are not.
Therefore, I shall define what an allergy is: an adverse reaction to a medication, food, or other substance that is mediated by the immune system and occurs only with exposure to the offending substance. People experience runny nose, itchy eyes, and sneezing with pollen exposure or more serious symptoms with consumption of certain foods. These are allergies because they involve IgE antibodies and immune cells like eosinophils, basophils, and mast cells. Other conditions may involve other components of the immune system with allergen exposure, like celiac disease with gluten consumption. Such conditions are technically allergic conditions because they involve adverse effects produced by the immune system.
What is not an allergy is everything else. If an adverse effect is not produced by the immune system itself, it's not an allergy. Of course, it's OK to tell your doctor that you once tried this medication and couldn't tolerate it because of a certain side effect. It may be an adverse reaction, but it doesn't necessarily mean it's allergic. Doctors ask patients about their allergy history because they don't want to put the patient at risk for severe allergy, the life-threatening type known as anaphylaxis.
Basically, any unwanted and/or unpleasant reaction to a food or drug is an adverse reaction. An allergy is just a specific type of it.