Part of the problem is that many Americans of non-Asian descent insist on treating "Asian" itself as one big ethnicity. An especially common variant of this trope is "All Asians Are Chinese" — which, again, is somewhat understandable, both because Chinese have been the most prolific Asian immigrant group in the United States as well as the majority of the world (mostly those from Guangdong in the West and those from Fujian in other parts of Asia). Admittedly, the other East Asian countries (Korea, Mongolia, Japan, and Vietnam... even if Vietnam is technically Southeast Asian) have been greatly influenced by the Chinese in their history, but that is where the similarity ends. Linguistically, East Asia cannot even be considered a single family; Chinese, Japanese, Korean, Mongolian and Vietnamese each belong to separate language families (Sino-Tibetan, Japonic, Koreanic, Mongolic, and Austroasiatic, respectively), so even in the distant past, they were already separate.
The problem is even bigger in Southeast Asia. Many countries in the region have big Chinese minority populations (the majority of Vietnamese and a large proportion of Thais have at least some Chinese ancestry, however distant, and of course, there's Singapore which is three quarters ethnic Chinese), but culturally they are a mixed bag. The Burmese, Thais, Lao, and Khmers owe their foreign influence more to India, from where they imported Buddhism and Brahmi script, though the Burmese language is a Sino-Tibetan one like Chinese. Indonesians, Malaysians and Bruneians also have a long history of Indian influence and are currently Muslim by religion. Filipinos are much the same, except that they have more European influence and are Christians, same with the East Timorese who have additional Melanesian influences. Don't even get started with language; 5 different families, including the aforementioned Sino-Tibetan and Austroasiatic. The languages of the Polynesians (Hawaiians, Maori, Samoans, and friends) are actually the eastern extension of the Austronesian languages spoken by the Filipinos, Indonesians, and Malaysians. This language family originated in Formosa before the island was colonised by Han Chinese settlers and renamed Taiwan with indigenous Formosans or Taiwanese Aboriginals now forming a minority of the population.
As for South Asia and the Middle East, many people ignorantly lump together Arabs, Persians, Turks, Indians, and Sri Lankans as one race of brown-skinned Muslims even though most Indians are Hindu (though the number of their Muslim minority easily dwarfs even some Muslim majority countries' population), most Sri Lankans are Buddhists, and many Arabs are Christian even if they pray in Arabic to Allah, since that's just the Arabic word for God. Religion aside, these are all distinct cultures with little in common, and not all Middle Easterners and South Asians even look like that, with many Caucasian-looking Middle Easterners and East Asian-looking South Asians existing. Arabic is of course linguistically close to Hebrew, while Persians, Indians, and Sri Lankans speak Indo-European languages, of the same stock as English, German, Russian, etc. Turks, meanwhile, speak a language that is alien compared to their neighbors; they immigrated from Central Asia to the Middle East during the 11th century, bringing along a culture that has a very much "far eastern" feel. A lot of the confusion here has to do with many people in the Arabian Peninsula being South Asian expats (especially laborers) with the United Arab Emirates and Qatar even being majority South Asian in population, South Asia indeed having several Muslim-majority countries in Afghanistan, Bangladesh, the Maldives and Pakistan, the two regions bordering each other with some definitions of the Middle East even including Afghanistan and Pakistan in it, and the two regions being the biggest conflict zones in Asia, the Middle East currently having the Israeli-Palestinian Conflict and the Syrian and Yemeni Civil Wars, and South Asia having the War in Afghanistan and The Indo-Pakistan Conflict, with both regions also dealing with problems of Islamic extremism. South Asia has also received Iranian influences through the Persian and Mughal Empires, with Afghanistan even speaking a dialect of Persian as one of their official languages along with another related language, Pashto, and the majority of their population being Iranian peoples, and Pakistan also using the Persian alphabet for their official language, Urdu, and having a significant Iranian minority.
And speaking of Central Asians, well, many people forget they even exist. Kazakhstan seems to be the sole exception, but even then they are generally seen as Borat stereotypes. Central Asia essentially acts as the boundary between the West and the East, with the region being located at the crossroads between Eastern Europe and Western, South, North and East Asia, having been influenced by Europe (especially Russia, with the entire region formerly being part of the Soviet Union), the Middle East (especially Iran) and East Asia (especially Mongolia), and was the former main hub of The Silk Road. They generally resemble a mix of Europeans and East Asians, and are culturally and ethnolinguistically related to Turks, with the exception of Tajikistan, who are instead related to Persians and Afghans (and are in fact more closely related to the original inhabitants of the region, as before the Mongoloid Turkic Peoples expanded from East Asia into the region, Central Asia was mainly inhabited by Caucasoid Eastern Iranians like the Bactrians, Dahae, Scythians and Sogdians, with one of the ethnic groups in Tajikistan, the Yaghnobi, being descended from the Sogdians).
Finally, it's often debated whether Russia itself, which encompasses a large part of Eastern Europe and the whole of North Asia, is a Western country which has territories in Asia (similar to British Overseas Territories) or an Asian country. On one hand, the majority of the country is in Asia, it is part of The Far East along with Southeast and East Asia (and is in fact the easternmost country in Asia), has notable Turkic, Mongolic and Tungusic minorities (as well as Yukaghir, Aleut and Eskimo peoples, the latter two notably also being found in Alaska, Canada and Greenland), has some Central and East Asian influences from it's neighbors and the native Siberian minorities, has had influence on Central, Southeast and East Asia and is a member of the Shanghai Cooperation Organisation along with China, India, Pakistan and Central Asia with the exception of Turkmenistan, but on the other hand, it's politically associated with Europe, its culture is a part of Western culture, its capital and largest city, Moscow, is in the European part of the country, the majority of the population lives in the European part of the country and the majority of the population are ethnic Russians, who are an East Slavic people and have a common ethnic and cultural origin with Belorussians and Ukrainians. The Caucasus finds itself in a similar situation, where they are in Western Asia like the Middle East and have been influenced by neighboring Turkey and Iran (with more Azerbaijanis living in Iran than in Azerbaijan itself), but are debated whether they are actually Asian or Europeans in Asia due to their culture (with Armenia and Georgia being heavily influenced by Orthodox Christianity) and politics.
This trope also exists in Britain, though with a difference (to American and Australian eyes, at least). The British use "Asian" as a reference to South Asians like Indians, Pakistanis, Bangladeshis and Sri Lankans. Immigrants from what, many years ago, was the British Indian Empire are the largest visible minority in the UK as are their British-born descendants. In the British census, "Asian" and "Chinese" are two different categories. People from the Far East (North, Southeast and East Asia) are not really called Asians; British people tend to refer to Chinese, Japanese, Siberians, Thai, etc. "Asians" are exclusively from the Indian subcontinent.