Is machine learning heuristic?

Table of Contents

Digital technology has disrupted several business industries, including entertainment, travel, banking, and electricity. Typical daily activities have become slow and outdated. These days cheques are deposited to bank accounts without visiting a local branch, products and services are purchased online, and food-service delivery apps deliver take-out food. Technology is creating data shared across multiple industries and sectors. An expert in one of these industries may find themselves working with vast amounts of complex data. Nowadays, computers solve incredibly complex problems. But to manage a problem, we should develop an algorithm. The human brain cannot accomplish such tasks.

However, exact algorithms might need years to manage with formidable challenges. In such cases, heuristic algorithms whose purpose is to seek approximate solutions but have adequate time and space complexity play an essential role. Different heuristic methods can help with complex data given limited time and resources.

Click this affiliate link to register for Machine Learning Using Python Training.

Click this affiliate link to order AI and Machine Learning for Coders:  A Programmers Guide to Artificial Intelligence.


Is machine learning heuristics

What are Heuristics?

The term “heuristic” is for algorithms that find adequate solutions among all available solutions, but they give no assurance that it is the best solution. Heuristic algorithms are considered as approximately and not accurate algorithms. These algorithms optimally find a solution close to the best one, and they find it quickly and easily. Sometimes these algorithms can be correct; that is, they find the perfect solution. The heuristic algorithm is an algorithm used until this best solution is proven to be the best. The method used from a heuristic algorithm is one known way, such as greediness, but the algorithm ignores or even suppresses some of the problem’s demands to be easy and fast. A heuristic algorithm does not guarantee an accurate solution. A good heuristic algorithm will get you an accurate enough solution. Heuristic techniques are used when there is no known solution algorithm or are interested in something faster than the known solution algorithm. 

For example, you see a set of bags on the one side of the room, and you have to select which weighs the heaviest. A reasonable heuristic would be to assume that the enormous bag is the heaviest one. The correct answer could be found by actually weighing the bags. However, it may be either that weighing the bags is impossible, or you do not want to spend the time to consider the bags. In such a scenario, you would use the heuristic method of guessing that the enormous bag would be the heaviest.

The critical point about a heuristic is that there is no way of knowing when the solution you get is wrong. If there were, you could create a self-correction loop and get the right answer, and that would mean you have a solution algorithm. But like with the bags, just by looking at them, you would never know when the largest is not the heaviest. With this bag weight heuristic, usually, and under a lot of conditions, you would be right. Using just your eyes to look at the bags, you could never know if the largest is full of pillows and the smallest is full of lead. You would never know when you were wrong. The best thing to do in practice is an experimental statistical study of the current situation and the appropriate answer.

Click this affiliate link to register for Machine Learning Using Python Training.

What do we mean by heuristics?

The word “heuristic” is gotten from the Greek word “eurisko,” which means “I find.” It has indicated a practical decision rule or a helpful way to find a solution to a problem in the past decades, relying upon experience and common sense. The aim of these rules isn’t to satisfy any formal or theoretical property; sometimes, heuristics algorithms have practical alternatives in contrast to standard mathematical algorithms. 

Heuristic algorithms are a way of problem-solving that uses shortcuts to produce close-enough answers given a specific time frame or deadline. Heuristic techniques are used for quick decisions, mainly when working with an intricate amount of data. Decision making using a heuristic approach may not necessarily yield the best results. Heuristics are shortcuts to solutions. The idea of heuristic methods in AI uses cognitive science or the study of how humans think. Indeed, people use heuristic algorithms all the time to make decisions and solve different problems. Heuristic algorithms are often used in artificial algorithms to get a computer system to find an approximate solution instead of an accurate solution.

Click this affiliate link to order AI and Machine Learning for Coders:  A Programmers Guide to Artificial Intelligence.

So why do we need heuristics? 

Given a specific time, one reason is to produce a good enough solution. It really does not have to be the most accurate one; an immediate answer will do since this is efficient enough. Most issues are exponential. This is used in artificial intelligence because we can use it in situations where we can’t find the best algorithms.

What are examples of heuristics?

Humans are more likely to see decisions as having higher rewards and lower risks when they are in a good mood. On the other hand, bad moods lead people to focus on the negatives rather than the possible benefits.

The gambler’s fallacy is a common heuristic. This fallacy entails thinking an outcome is due to occur after a long stretch of its absence. For example, people often feel a coin is bound to land tails-up if it has landed heads-up five times in a row. In reality, an unbiased coin is equally likely to land on either side, no matter its history. 

A kind older looking man might remind you of your father, so you might quickly assume that he is friendly, peaceful, and cheerful.

While sitting and waiting at the airport for your flight, you might find yourself thinking about previous airplane accidents, asking yourself to take a car instead. The disasters are horrible, making traveling by air seem more dangerous and prevalent than it is.

Click this affiliate link to register for Machine Learning Using Python Training.

What is a Heuristic Search?

A heuristic search is a technique to solve a problem faster than classic methods or find an approximate solution when traditional methods cannot. This heuristic technique is a shortcut as we exchange optimality, completeness, accuracy, or precision for speed. A heuristic function considers different search algorithms. At descending branching steps, it evaluates the information available and chooses which branch to follow. It does so by ranking several alternatives. The heuristic search is any device that is often efficient but will not guarantee the right solution in every case.

Click this affiliate link to order AI and Machine Learning for Coders:  A Programmers Guide to Artificial Intelligence.

Heuristic Search Techniques in Artificial Intelligence

Briefly, we can classify heuristic techniques into two categories:

1. Direct Heuristic Search Technique in AI: This technique goes by other names such as Blind Search heuristic, Uninformed Search heuristic, and Blind Control Strategy. These techniques are complicated and almost impossible because they demand much time or memory. They undergo a process of searching the entire state space for a solution and use an arbitrary ordering of operations. Some examples of these include Breadth-First Search (BFS) and Depth First Search (DFS).

2. Weak Heuristic Search Technique in AI: This technique goes by other names such as Informed Search, Heuristic Search, and Heuristic Control Strategy. These are very efficient and effective when applied correctly to the right types of tasks. Weak, heuristic search techniques demand domain-specific information. This additional piece of information is needed to compute preference and expand the different child nodes. For each node, there is a heuristic function associated with it. Some examples are Best First Search (BFS) and A* search.

Listed below are some generally observed search techniques. 

· Best-First Search

· A* Search

· Bidirectional Search

· Tabu Search

· Beam Search

· Simulated Annealing

· Hill Climbing

· Constraint Satisfaction Problems

Click this affiliate link to register for Machine Learning Using Python Training.

Hill Climbing in Artificial Intelligence

Let us discuss Hill Climbing in Artificial Intelligence. This technique is a heuristic for optimizing problems mathematically. To maximize or minimize a real function, we have to choose values from the input. There is no problem if the solution isn’t the global optimal maximum.

An example of the Hill Climbing technique widely discussed is the Travelling Salesman Problem. It is a problem where we must minimize the distance the salesman travels.

Features of Hill Climbing in AI

• It is a type of the generate-and-test algorithm.

• It makes use of the greedy approach. This approach keeps generating possible solutions until it finds the expected answer and moves only in the direction that optimizes the cost function.

Click this affiliate link to order AI and Machine Learning for Coders:  A Programmers Guide to Artificial Intelligence.

Types of Hill Climbing in AI

Simple Hill Climbing- This is the process of examining a neighboring node at a specific time and selecting the fastest node that best optimizes the current cost to be the next node in line.

Steepest Ascent Hill Climbing- This is the process of examining all neighboring nodes and picking the one closest to the solution area.

Stochastic Hill Climbing- This is the process of picking a neighboring node at random and deciding whether to use it or look at another.

Constraint Satisfaction Problems (CSP)

A constraint can easily be nothing but a limitation or a restriction. Working with AI, some specific conditions are necessary for problems to be solved. Let’s solve a problem, shall we? Let’s talk of a magic square. It is a sequence of numbers- usually integers- arranged in a square grid. Each diagonal, column, and row consecutively all add up to a constant called the Magic Constant.

Simulated Annealing Heuristic Search: 

In metallurgy, when we ‘slow-cool’ metals to pull them down to a low energy state, it gives them outstanding amounts of strength. We call this annealing. It is a process of high temperatures observing much random movement, while low temperatures keep microscopic randomness. Artificial intelligence takes a cue from the field of metallurgy to produce the heuristic search called simulated annealing. It is a process of optimization that begins with a random search at a very high temperature. After that, the temperature slowly reduces. Eventually, as the temperature reaches zero, the search becomes pure greedy descent. At each step, this simulated annealing heuristic search randomly chooses a variable and a value. It takes the assigned value only when it leads to an improvement or doesn’t lead to more conflict. Else checks if the temperature is much worse than the current project to accept the position with some probability. An annealing schedule defines how far the temperature drops as the search progresses. A ubiquitous program is geometric cooling. 

Best-First Search (BFS) Heuristic Search: 

It is also called BFS. Best First Search is an informed heuristic search that chooses an evaluation function to select which adjacent is the most promising before exploring. Breadth- and Depth- First Searches uses a blind approach to explore paths without keeping a cost function in mind. Things aren’t the same with BFS, though. A priority queue is a function that stores node costs. We can understand the BFS Heuristic Search through the pseudocode below.

Step 1 = Define list OPEN with single node s– the start node.

If the list is empty, return failure.

Step 2 = Remove node n (node with the best score) from the list, move it to list CLOSED.

Step 3 = Expand node n.

Step 4 = If any successor to n is the goal node, return success, and trace the path from goal node to s to produce the optimal solution.

Step 5 = For each successive node: we apply the evaluation function, f. If the node isn’t in either list, add it to the list OPEN.

Step 6 = Loop to step 2 and run again.

Click this affiliate link to register for Machine Learning Using Python Training.

Heuristic techniques 

Branch-and-bound technique and dynamic programming are quite useful, but their time-complexity often is too high and unacceptable for NP-complete tasks. The hill-climbing algorithm is practical, but it has a significant drawback called premature convergence. Since the algorithm is “greedy,” it always finds the closest local optima with the lowest quality. Modern heuristics aims to overcome this disadvantage. 

Simulated annealing algorithm, initially used in 1983, uses an approach similar to hill-climbing, but frequently receives worse solutions than the current. The probability rate of acceptance decreasing with respect to time. 

Tabu search uses the idea of memory structures to avoid local. The major problem with the simulated annealing heuristic technique is that after “jump,” the algorithm is likely to repeat its track. Tabu search prohibits the repetition of moves made recently. 

Swarm intelligence is an artificial intelligence technique introduced in 1989. It is the study of collective behavior in decentralized, self-organized systems. Under Swarm intelligence, two of the most successful types approach are:

• Ant Colony Optimization (ACO) 

• Particle Swarm Optimization (PSO). 

1. In ACO, artificial ants build solutions by moving on the problem graph and changing it so that future ants can develop better solutions. 

2. PSO takes the problem-solving approach of finding the best solution, representing a point or surface in an n-dimensional space. 

The significant upside of swarm intelligence techniques is that they are highly resistant to local optima. 

Evolutionary algorithms aim to succeed in tackling premature convergence by considering different solutions at the same time. 

Biological neuron systems inspire the Neural Networks approach in machine learning. They consist of units, called neurons, with interconnections between them. After training occurs on a given data set, Neural Networks can make predictions from new data, not in the training set. In practice, Neural Networks do not always work well because they suffer significantly from problems of underfitting and overfitting. These problems will have a direct bearing on the accuracy of prediction. If a network is not involved, it may simplify the laws which the data learns from over a specific time. From the other school of thought, if a system is very complex, it can consider the noise that usually helps in the training data set while assuming the laws. The quality of prediction deteriorates after training in both cases. Neural Networks suffer from the problem of premature convergence. 

Support Vector Machines is an extension of the ideas of Neural Networks. The convex objective function aims to overcome premature convergence; therefore, only one optimum exists. The support vector machine, which is also known classically as the divide and conquer technique, gives an efficient solution for separable problems. Support vector machines provide useful classification techniques that become a mighty instrument. 

Click this affiliate link to order AI and Machine Learning for Coders:  A Programmers Guide to Artificial Intelligence.

Does machine learning have heuristic?

A heuristic is usually a hand-coded function. It is not a machine learning model obtained by training on a data set but embodies some common-sense features derived from domain experts.

A Machine Learning algorithm learns from previous data. For example, an online learning algorithm is one that keeps adapting its model for each example it sees. The longer the usage, the better it gets (depending on the class of models used is sufficient for the task).

A heuristic does not change, but it remains fixed. One neat trick is that we can use both a heuristic in a machine-learning algorithm to focus on some promising candidate models first. This process significantly speeds up the learning process, or Machine Learning is used to learn a simple but fast model useful heuristic.

You look for differences between two entities only if there are similarities between them. So, let us first look at the similarities between Machine Learning and Heuristics.

Machine Learning and Heuristics are both iterative methods to solve optimization problems. These optimization problems can be:

The shortest distance traveled between cities given all cities are visited at least once

finding optimal parameters which lead to the smallest cost function

finding global maxima or minima of a polynomial

Some examples of heuristics are:

Simulated Annealing process for solving a complex optimization problem like Travelling Salesman Problem

Newton Raphson method to solve for roots of a polynomial

Some examples of machine learning algorithms are:

In a neural network-based method, finding optimum weights defining the non-linear relationship between the predictor and outcome variables using stochastic gradient descent method for creating strong learner models from weak learner models by assigning higher values to examples with high error, e.g., in Adaboost or Xgboost.

Click this affiliate link to register for Machine Learning Using Python Training.

The advantages and disadvantages of the heuristic approach

Heuristics facilitate timely decisions. Analysts in every industry use techniques such as intelligent guesswork, trial and error, the process of elimination, using past formulas, and the analysis of historical data to solve a problem. Heuristic methods make decision-making straightforward and efficient through short cuts and close-enough calculations.

The significant advantage of adopting a heuristic approach is that it offers a quick solution that is easy to understand and implement. Heuristic algorithms are efficient. It serves as fast and practical short-term solutions to forecasting and scheduling problems.

The heuristic approach’s main downside is that it cannot deliver an optimal solution to a planning and scheduling problem in most cases.

Heuristic search techniques use a quick fix to specific planning or scheduling issues but cannot serve as viable solutions that deliver the best-needed outcomes. Heuristic techniques tend to “leave money on the table” – they often stop with a solution, even though there are better solutions to the same problem that yield lower supply chain cost, higher-order satisfaction performance, or an increase in total profit. In the long run, as your business model and processes grow and expand, heuristic techniques will provide solutions that will inevitably fail – as they are complex and can’t accommodate your company’s growing needs and requirements.

Click this affiliate link to order AI and Machine Learning for Coders:  A Programmers Guide to Artificial Intelligence.

Another major disadvantage is the lack of flexibility that the heuristic techniques possess. Suppose a company wants to add a new set of machines to the production line; it creates a shift in the bottle-neck production process. In that case, a hard- or pre-coded heuristic may no longer serve as a valid and viable solution. A simple change in the operational procedures or a difference in the underlying data structures, such as the distribution of demand over time or product mix, can have a significant impact on the performance of the heuristic algorithm – and this can pose a severe risk to your company’s overall productivity and profitability.

There are trade-offs using heuristics that render the approach prone to bias and errors in judgment. The final decision of the user may not be the optimal or accurate solution. The decision made may be inaccurate. The data used might be messy and unstructured, thereby providing an incorrect answer to the initial problem. Copycat investors usually imitate successful investment managers’ characteristics, copying their investment patterns and techniques to avoid researching securities and the associated research, quantitative and qualitative information on their own.

Heuristic techniques are practical and provide efficient and feasible short-term solutions to forecasting and predicting problems, but lack the efficiency and flexibility to create long-lasting, optimal solutions that create pathways to greater productivity and profitability.

Click this affiliate link to register for Machine Learning Using Python Training.

Luis Gillman
Luis Gillman

Hi, I Am Luis Gillman CA (SA), ACMA
I am a Chartered Accountant (SA) and CIMA (SA) and author of Due Diligence: A strategic and Financial Approach.

The book was published by Lexis Nexis on 2001. In 2010, I wrote the second edition. Much of this website is derived from these two books.

In addition I have published an article entitled the Link Between Due Diligence and Valautions.

Disclaimer: Whilst every effort has been made to ensure that the information published on this website is accurate, the author and owners of this website take no responsibility  for any loss or damage suffered as a result of relience upon the information contained therein.  Furthermore the bulk of the information is derived from information in 2018 and use therefore is at your on risk. In addition you should consult professional advice if required.