Police say the bodies of nine women and four men were found washed up on a beach in Johor state early on Tuesday.
Those on board were thought to be illegal migrants or workers from Indonesia, they added.
Thousands of Indonesians work illegally in plantations and other industries in Malaysia.
They often risk dangerous sea journeys to bypass border controls when they enter Malaysia or return home.
District police chief Rahmat Othman said he believed up to 35 people could have been on board the boat, and a search and rescue operation was under way.
An Indonesian foreign ministry spokesman told Reuters that "ID cards from Indonesia" were found on the bodies of the 13 victims.
Accidents off Malaysia's coast are relatively common as illegal migrants try to find work in the relatively affluent country.
It is also a key transit hub for asylum seekers fleeing unrest in their own countries, such as in Myanmar, also known as Burma.
Malaysia has its origins in the Bangla kingdoms present in the area which, from the 18th century, became subject to the British Empire.
The first British territories were known as the Straits Settlements, whose establishment was followed by the Malay kingdoms becoming British protectorates.
The territories on Peninsular Malaysia were first unified as the Malayan Union in 1946.
Malaya was restructured as the Federation of Malaya in 1948, and achieved independence on 31 August 1957.
Malaya united with North Borneo, Sarawak, and Singapore on 16 September 1963. Less than two years later in 1965, Singapore was expelled from the federation.
The country is multi-ethnic and multi-cultural, which plays a large role in politics. The constitution declares Islam the state religion while allowing freedom of religion for non-Muslims.
Since its independence, Malaysia has had one of the best economic records in Asia, with its GDP growing at an average of 6.5% per annum for almost 50 years.
The economy has traditionally been fueled by its natural resources, but is expanding in the sectors of science, tourism, commerce and medical tourism. Today,
Malaysia has a newly industrialized market economy, ranked third largest in Southeast Asia and 29th largest in the world.