Since 2011, the proliferation of brown macro-algae of the genus Sargassum has considerably increased in the North Tropical Atlantic Sea, all the way from the Gulf of Guinea to the Caribbean Sea and the Gulf of Mexico. The large amount of Sargassum aggregations in that area cause major beaching events, which have a significant impact on the local economy and the environment and are starting to present a real threat to public health. In such a context, it is crucial to collect spatial and temporal data of Sargassum aggregations to understand their dynamics and predict stranding. Lately, indexes based on satellite imagery such as the Maximum Chlorophyll Index (MCI) or the Alternative Floating Algae Index (AFAI), have been developed and used to detect these Sargassum aggregations. However, their accuracy is questionable as they tend to detect various non-Sargassum features. To overcome false positive detection biases encountered by the index-thresholding methods, we developed two new deep learning models specific for Sargassum detection based on an encoder–decoder convolutional neural network (CNN). One was tuned to spectral bands from the multispectral instrument (MSI) onboard Sentinel-2 satellites and the other to the Ocean and Land Colour Instrument (OLCI) onboard Sentinel-3 satellites. This specific new approach outperformed previous generalist deep learning models, such as ErisNet, UNet, and SegNet, in the detection of Sargassum from satellite images with the same training, with an F1-score of 0.88 using MSI images, and 0.76 using OLCI images. Indeed, the proposed CNN considered neighbor pixels, unlike ErisNet, and had fewer reduction levels than UNet and SegNet, allowing filiform objects such as Sargassum aggregations to be detected. Using both spectral and spatial features, it also yielded a better detection performance compared to algal index-based techniques. The CNN method proposed here recognizes new small aggregations that were previously undetected, provides more complete structures, and has a lower false-positive detection rate.