Scraping a web page on-the-fly to retrieve public data

Image for post
Image for post
Photo by Michał Parzuchowski on Unsplash

Many online services do not offer APIs to give access to their public data. At the same time, they might have all this data available on their website. In such a circumstance, why not scrape it?

Web scraping is a complicated subject and — to perform it consistently — you may need an equally complex solution. If this is your case, I recommend this article, where I delved into building a robust, advanced, and modern rotating IP and User-Agent web scraping script.

In most cases, you do not need such a sophisticated system. In fact, an API that is capable…


Handle different MongoTemplates with the MongoDB 4.x driver

Code
Code
Photo by AltumCode on Unsplash.

There are plenty of online guides explaining how to configure a Spring Boot application that connects to multiple MongoDBs. The common approach was to use MongoDbFactory, but this class was deprecated in Spring Data MongoDB 3.x, which introduced the MongoDB 4.x driver.

So, let’s see how to configure multiple MongoDB data sources in Spring Boot using Spring Data MongoDB 3.x.

Please note that code will be written in Kotlin, but the same result can be achieved in Java as well. …


Making your API layer more robust and reliable

Image for post
Image for post
Photo by Maxwell Nelson on Unsplash

In my previous article, I showed how to define an API Layer to avoid code duplication in Spring Boot.

Its idea is based on separating where all your needed API requests are defined from where they are actually called. Having all the definitions in one place allows you to easily add shared custom behavior, such as retry logic.

After all, your back-end probably depends on other services. What would you do when they fail? Obviously, this is not your fault, but at the same time, you cannot expect users to restart their entire workflow due to avoidable errors.

Even if…


Configure Sequelize and Mongoose to handle concurrent data-intensive requests

Code on a laptop
Code on a laptop
Photo by Joshua Aragon on Unsplash.

Recently, I dealt with a huge performance problem in Node.js. My backend built using Sequelize and Mongoose was often unable to respond to any requests for minutes.

Let’s see what I did to identify the bottlenecks, why such issues can happen to any Node.js developer, and how to avoid them.

Detecting the Bottlenecks

I had monitored my VPSs for days without noticing anything alarming before realizing that the problem was elsewhere. Each of the databases used by the application has its own VPS. Data is very large — hundreds of millions of records — but the 16GB 4 CPU VPSs I am using…


Security as a key to business success in IT

Image for post
Image for post
Photo by Franck on Unsplash

DevOps has been one of the most important tech trends of the last 5 years and we have all heard about it at least once. While we should all know what DevOps is, no one really knows how it will look like in the future.

Not surprisingly, DevOps has evolved incredibly since its appearance and DevSecOps represents its natural evolution.

In this article, I am going to introduce DevSecOps and explain why it is one of the most important and interesting technology trends in 2021.

What is DevOps?

Although this is not the main goal of the article, before diving into DevSecOps, spending…


Avoiding conversion overhead by returning CSV data directly

Image for post
Image for post
Photo by Matthew Fournier on Unsplash

Importing data from CSV (Comma-Separated Values) files is a basic feature offered by many different services. At the same time, CSV is one of the most popular formats in data science.

Although dealing with CSV content is common, the approach I used to follow was to convert the data returned by an API or a query into a valid CSV file. This involves boilerplate code and unnecessary overhead.

So, producing CSV as output, allowing users to download it, is a more practical, advanced, and automatable solution.

Let’s see how to build an API returning CSV content in Spring Boot.

1. Adding the Required Dependencies

There…


Witnessing a dramatic story taught me its meaning the hard way

I will never forget 2020, but 2021 has been even worse so far. I can affirm this because I spent the last few weeks living in a surreal atmosphere. My friend, and neighbor, spent over twenty days in the hospital, struggling between life and death against a severe case of Covid. Although having enough energy to leave his home on his own two feet, this was not enough.
That was his last journey.

Image for post
Image for post
Photo by Casey Fyfe on Unsplash

Coping With Emotions

In his last weeks on Earth, the doctors told her wife that although they were trying everything, there was no real hope. …


How the Divine Comedy inspired my academic success

Image for post
Image for post

I was one of the top students in Italy and I have always been a Dante enthusiast. The Divine Comedy was the main source of motivation during my academic years. The insights contained in that poem are so universal there is no field in which they cannot be applied.

Many students perceive their academic life as a strenuous challenge.

After all, Dante wrote:

The path Paradise begins in Hell

So why not comparing an academic career with the extraordinary journey into the underworld described by Dante?

Let’s dive into the 9 keys to academic success inspired by the Divine Comedy.

Hell


No longer get lost in your backend

Image of laptop with code.
Image of laptop with code.
Photo by Goran Ivos on Unsplash

Working with backends, I learned how you can easily lose sight of what is going on or what is currently deployed. This might become a problem, especially when dealing with production environments.

There are many libraries to tackle this question, and I personally recommend Spring Boot Actuator.

“Spring Boot includes a number of additional features to help you monitor and manage your application when you push it to production. You can choose to manage and monitor your application by using HTTP endpoints or with JMX. Auditing, health, and metrics gathering can also be automatically applied to your application.” …

Antonello Zanini

Technology Bishop, Freelance Software Engineer, Writer ‎‏‏https://antonellozanini.com/en

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store