DEV Community: ChelseaLiu0822

PySpark： missing value

ChelseaLiu0822 — Thu, 18 Apr 2024 04:41:44 +0000

Drop

df.na.drop() vs. df.dropna()

DataFrame.dropna() and DataFrameNaFunctions.drop() are aliases of each other. So theoretically their efficiency should be equivalent.

In addition, df.na.drop() can also specify a subset.

examples

# Code to drop any row that contains missing data
df.na.drop().show()

# Only drop if row has at least 2 NON-null values
df.na.drop(thresh=2).show()

# Only drop the rows with null in Sales col
df.dropna(how='any',subset='Sales').show()

df.na.drop(how='any').show()
df.na.drop(how='all').show()

fill

We can also fill the missing values with new values. If you have multiple nulls across multiple data types, Spark smart enough to match up the data types. For example:

df.na.fill('NEW VALUE').show()

if you have multiple columns to fill, you could use a dictionary.

JUC

ChelseaLiu0822 — Thu, 30 Nov 2023 06:14:23 +0000

AQS principle

AQS (ABstractQueuedSynchronizer) is a framework for blocking locks and related synchronizer tools
Features:
Use the state attribute to represent the status of the resource (exclusive and shared). The subclass needs to define how to maintain this status.
Control how locks are acquired and released
getState - Get the state
setSate - set state
compareAndSetState - Optimistic lock set state
Exclusive: only one thread can access Shared: multi-threaded access
Provides a FIFO-based waiting queue, similar to Monitor's EntryList
Condition variables are used to implement waiting and wake-up mechanisms, and support multiple condition variables, similar to Monitor's WaitSet.
tryAcquire() acquires the lock
tryRelease() releases the lock
The bottom layer uses park and unpark to block and wake up threads

ReentrantLock principle

Principle of unfair lock implementation:

Threads that fail in competition will enter the Node queue. The waitStatus of Node is 0 by default. The creation of Node is lazy.
Yes, the first Node is Dummy or Sentinel, which is used to occupy space and is not associated with threads.
acquireQueued: After entering Node, Thread-1 will continuously try to acquire the lock in an infinite loop and fail.
After entering park blocking, if the previous node of the current Node is head, its status will be changed to -1, indicating that it is responsible for calling
Wake up a node and try to acquire the lock again. If it fails, block the thread.
Lock competition is successful:
No race conditions:
After the lock is released, check whether waitSet is null and whether the node status is -1, find a Node closest to the head, unpark resumes its operation, and updates the head node.
There is competition: If the lock is obtained by another thread first, enter the acquireQueued process again. If it fails, re-enter the park blocking process.
Re-entry principle:
Determine whether the current incoming thread is an occupying thread and count state+1. When releasing, first state -1 and only state=0 can be released.
Interruptible principle:
Uninterruptible mode (default). Even if it is interrupted, it will still stay in the AQS queue. After obtaining the lock, it will continue to run (the interrupt flag is set to true) to know that it has been interrupted and interrupt. Cannot be interrupted in blocking state
Interruptible mode: acquireInterruptibly() method directly throws an exception, leaves AQS, and stops waiting for the lock.

Fair lock:
During the competition, it will be checked to see if there is a predecessor node. If there is no predecessor node, it will compete.
Condition variable implementation
await process

Then enter the fullyRelease process such as AQS, release the lock on the synchronizer, directly release the reentry lock, and unpark the next node in the AQS queue to compete for the lock.

signal
First determine whether it is a thread holding the lock for signaling. If so, get the first node of the waiting queue, transfer this node to the end of the AQS queue, and convert its own status to 0, and set the previous node status to - 1

read-write lock

ReentrantReadWriteLock
When read operations are much higher than write operations, use read-write locks to allow read-read concurrency and improve performance.
Usage: Need to obtain read lock and write lock respectively

Read locks do not support condition variables
Upgrading during re-entry is not supported: Acquiring a write lock when there is a read lock will cause the acquisition of the write lock to wait forever.
It can be reduced when re-entering, first acquire the write lock, and then acquire the read lock.
Cache update strategy
Clear the cache first, then update the database
Read-write lock principle
The same Sync synchronizer is used, so the waiting queue and state are also the same.

t2 executes r.lock, and then enters the sync.acquireShared(1) process of the read lock. It will first enter the tryAcquireShared process. If there is a write lock occupied, then -1 is returned to indicate failure.
Returning 0 indicates success, but subsequent nodes will not continue to wake up.
Returning a positive integer indicates success, and the value indicates that there are still several successor nodes that need to be awakened. The read-write lock returns 1

Execute tryAcquireShared again. If successful, increase the read lock count by 1.

The wake-up of the read lock will wake up all shared threads to achieve read-read concurrency.
StampedLock
It was added in JDK8 to further optimize the reading performance. Its characteristic is that when using read locks and write locks, it must be used in conjunction with [stamp].

Optimistic reading, StampedLock supports the tryOptimisticRead() method (no lock is used in this method). After reading, a [stamp verification] needs to be done. If it passes, it means that there was no writing operation during the period and the data can be used safely. If it fails, you need to re-obtain the read lock to ensure data security.

StampedLock does not support condition variables and reentry.
Semaphore
Semaphore, used to limit the number of threads that can access shared resources at the same time

*CountdownLatch *(countdown lock)
Used for thread synchronization and cooperation, waiting for all threads to complete the countdown
The construction parameters are used to initialize the wait count value, await() is used to wait for the count to return to zero, and countDown() is used to decrement the count by one.

CyclicBarrier
Loop fence is used for thread collaboration, waiting for threads to meet a certain count. The number of counts is set during construction. When each thread executes to a certain moment that requires synchronization, the await() method is called to wait. When waiting When the number of threads reaches the count, continue execution.

The number of threads needs to be the same as the count value
It can be used repeatedly without creating a new CyclicBarrier object. After the count is used, it will be reset to the initial value. CountdownLatch is a one-time use.

Thread safe collection class

ChelseaLiu0822 — Thu, 30 Nov 2023 05:59:43 +0000

Blocking: Based on locks, through blocking
CopyOnWrite: modification overhead is high
Containers of Concurrent type:
Through cas optimization, high throughput
Weak consistency: When traversing, even if the container is modified, the iterator can continue to traverse, and the content is old
Seek consistency between sizes and sizes. The size operation may not be 100% accurate.
Read weak consistency
If modifications occur during traversal of a non-thread-safe container, the fail-fast mechanism will be used to cause the traversal to fail and throw
ConcurrentModificationException, no longer traversed.

ConcurrentHashMap
Same as the normal HashMap method, but thread-safe
Although each operation is atomic, the combination is not necessarily thread-safe.
map.computeIfAbsent() puts the key into the map. It is thread-safe and ensures the atomicity of the get and put method combinations.
principle:
HashMap Java 8 hash conflict elements are placed at the end of the linked list, while in Java 7 they are placed at the head.
The problem of concurrent dead links will occur when HashMap is expanded (occurred in JDK 7). The insertion method has been modified in JDK 8 so that dead links will no longer be formed during expansion, but it will easily lead to data loss during expansion.
ConcurrentHashMap

Lazy initialization, only the table size is calculated in the construction method, and it is not actually created until it is used for the first time.
The structure used in JDK8 is array + linked list + red and black. The initial length of the array is 16
When the length of the linked list exceeds 8, it turns into a red tree. When the length exceeds the loading factor * array length, the capacity is expanded.
In JDK7, Segment segment lock is used to achieve thread safety based on array + linked list.
Using CAS+Synchronized in JDK 8 to ensure thread safety

Thread Pool

ChelseaLiu0822 — Thu, 30 Nov 2023 05:55:47 +0000

ThreadPoolExecutor

Thread pool status:
ThreadPoolExecutor uses the high 3 bits of int to represent the thread pool status, and the low 29 bits represent the number of threads.

This information is stored in an atomic variable ctl.

When the task volume is large, an emergency thread will be created to execute the task, and it will be destroyed after execution.

RejectedExecutionHandler

newFixedThreadPool

Number of core threads == Maximum number of threads (no emergency threads), no timeout required
Suitable for situations where you know the number of tasks and are not too busy
Because it is a core thread, it will not end the thread by itself after executing the task.

newCachedThreadPool (thread pool with buffering function)

The number of core threads is 0, the maximum number of threads is Integer.MAX_VALUE, all created are emergency threads, and the lifetime is
It is 60s. Emergency threads can be created unlimitedly.
The queue is implemented using synchronousQueue and has no capacity limit.

newSingleThreadExecutor

Applicable to situations where multiple tasks are expected to be executed in queue order. Even if an exception is encountered, a new thread will be created to execute the remaining tasks.

Close thread pool
The thread pool status changes to SHUTDOWN and the submitted task is executed without blocking the execution of the calling thread.
void shutdown()
The shutdown() method will modify the thread pool status, interrupt idle threads, and attempt to terminate

shutdownNow()
The thread pool status changes to STOP, and new tasks will not be accepted. The tasks in the queue will be returned, and the interrupt method is used to interrupt the executing tasks.
The shutdownNow() method will modify the thread pool status, interrupt all threads, obtain the remaining tasks in the queue, and try to terminate it.

Design Pattern Worker Thread

Let limited worker threads take turns to process an unlimited number of tasks asynchronously
Applications of different task types use different thread pools, which can avoid starvation and improve efficiency.

hunger Fixed-size thread pools may experience starvation The threads in the same thread pool all do the same task, and there are no remaining threads to do the following tasks, leading to starvation. Appear Let different thread pools do different tasks
Thread pool size If it is too small, the thread cannot make full use of system resources and can easily lead to starvation. Excessive size leads to more thread context switching and takes up more memory. Determine the thread pool size according to different situations
CPU-intensive operations (e.g. data analysis) Using CPU core number +1 can achieve optimal CPU utilization. +1 ensures that when the thread is suspended due to page missing failure or other problems, the additional thread can step in to ensure that the CPU clock cycle is not wasted.
IO intensive CPU is not always busy Empirical formula: Number of threads = number of cores * expected CPU utilization * total time (CPU calculation time + waiting time) / CPU calculation time
Task scheduling thread pool It can be implemented using Timer, which is simple and easy to use. However, all tasks are scheduled by one thread, so the tasks are executed serially. The delay or exception of one task will affect the subsequent tasks. Use newScheduledThreadPool instead. schedule (task, time, time unit) method to delay execution Construction method:

Thread pool exception Exceptions will not be printed, and you need to try catch to catch them manually. The exception will be caught and printed through the get() method of the Future object
Tomcat

LimitLatch is used to limit current and control the maximum number of connections.
Acceptor receives new socket connections
Poller monitors the socket channel to see whether there are readable IO events. Once it is readable, it encapsulates a task object (socketProcessor) and submits it to the Executor thread pool for processing.
The worker threads in the Executor thread pool are responsible for processing requests
Tomcat thread pool extends ThreadPoolExecutor. If the total thread reaches maximumPoolSize, RejectExecutionException will not be thrown immediately. It will try to put the task into the queue again. If it fails, RejectExecutionException will be thrown.

Fork/Join thread pool The idea of divide and conquer is used, which is suitable for CPU-intensive operations of task splitting. Multi-threading is added on the basis of divide and conquer, and the decomposition and merging of each task is handed over to different threads to complete. Step by step to improve computing efficiency By default, it is created in a thread pool with the same number of CPU cores. How to use: The task object needs to inherit RecursiveTask and override the compute() method

Shared model immutability

ChelseaLiu0822 — Thu, 30 Nov 2023 05:43:47 +0000

SimpleDateFormat is thread-unsafe, you can use DateTimeFormatter (immutable class, thread-safe) under multi-threading

Immutable design:

Classes and all properties in a class are final Modifying the property with final ensures that the property is read-only and cannot be modified. Modifying a class with final ensures that methods in the class cannot be overridden, preventing subclasses from inadvertently destroying immutability. -Protective copy: avoid sharing by creating copy objects Protective copy to ensure that the array contents will not be changed by other classes.

Flyweight pattern

Share data as much as possible
Since protective copy creates multiple objects and wastes resources, flyweight mode is usually used.
This pattern is often used in wrapper classes
The valueOf method of Long type will cache Long objects between -128 and 127. Objects will be reused within this range. New Long objects will be created only when the range is greater than this range.

final principle

Principle of setting final variables

Stateless is also thread-safe and is achieved by not setting member variables in the class.

Lock-free concurrency-optimistic locking (non-blocking)

ChelseaLiu0822 — Thu, 30 Nov 2023 05:40:40 +0000

CAS and volatile
CAS: compare And Set (compare And Swap), must be an atomic operation

The bottom layer of CAS is the lock cmpxchg instruction (X86 architecture), which ensures operation under both single-core CPU and multi-core CPU.
atomicity
CAS operations require volatile support to ensure that the latest results of shared variables are obtained to achieve [compare and exchange]Effect.

High lock-free efficiency

In a lock-free situation, even if the retry fails, the thread is always running at high speed without stopping, and synchronized will make
When the thread does not acquire the lock, a context switch occurs and it enters blocking.
In the lock-free case, because the thread needs to keep running, additional CPU support is required. If the thread is not allocated time,
file, it will still enter the runnable state, and it will still cause context switching.

CAS features

CAS is based on the idea of optimistic locking: it is not afraid of other threads modifying shared variables.
synchronized is based on the idea of pessimistic locking: preventing other threads from modifying shared variables
CAS embodies lock-free concurrency and non-blocking concurrency:
Synchronized is not used, so the thread will not be blocked.
However, if competition is fierce and retries occur frequently, efficiency will be affected.

Atomic integer

AtomicBoolean, AtomicInteger, AtomicLong basic types of atoms
Construction method:
AtomicInteger i = new AtomicInteger(value) value is modified by volatile
Auto-increment method:
i.incrementAndGet()i++
i.getAndIncrement()++i
Similar to autodecrement
Get value:
i.get()
addition:
i.getAndAdd(value) i.addAndGet(value)
multiplication:
i.updateAndGet (IntUnaryOperator operation) (IntUnaryOperator is a function interface)
i.getAndUpdate
e.g i.updateAndGet(x -> x*10)
principle:

Atomic citation

AtomicReference, AtomicMarkableReference, AtomicStampedReference
AtomicReference determines the protected object through generics
Basically the same usage as AtomicInteger

ABA question:

The initial variable is A. Before CAS, other threads changed the shared variable to B, and another thread changed B to
A. CAS cannot sense whether the shared variable has been modified. CAS can only determine whether the shared variable is the same as the original value.

AtomicStampedReference

By adding the version number attribute to the shared variable, we can determine whether the shared variable has been modified. If the CAS has been modified,
will fail
Construction method:
AtomicStampedReference ref = new AtomicStampedReference<>("A",0(version
This number))
There is no get() method, only getReference() method
compareAndSet(prev, new, stamp, newStamp), you need to pass in the original version number, and
Modified new version number

AtomicMarkableReference

Only care whether the shared variable has been changed, not how many times it has been changed, just add boolean to judge
Construction method:
AtomicMarkableReference ref = new AtomicMarkableReference<> (value,
true)
compareAndSet(prev, new, expectedMark, newMark) needs to pass in the original mark and modify next new tag

Atomic array

AtomicIntegerArray, AtomicLongArray, AtomicReferenceArray
Protect the thread safety of elements within an array
Usage is basically the same as AtomicInteger
The length() method returns the array length
getAndIncrement(index) auto-increments the elements in the array

Atomic field updater

Protect object member variables from thread safety
AtomicReferenceFieldUpdater, AtomicIntegerFieldUpdater,
AtomicLongFieldUpdater
Construction method:
AtomicReferenceFieldUpdater updater = new AtomicReferenceFieldUpdater(protect
Object.class, reference object.class, member variable)
compareAndSet(object, initial value, expected value)
The member properties of the object must be modified with volatile

Atomic Accumulator

More efficient than the self-increment method in AtomicLong
LongAdder
increment() self-increment method
reason:
When multiple threads compete, CAS competition becomes fierce, resulting in low efficiency. When there is competition, LongAdder sets multiple accumulation units, and different threads accumulate different units (Cell variables), which reduces CAS retry failures, thereby improving performance. efficiency

Unsafe object

Provides very low-level methods for operating memory and threads. Unsafe objects cannot be called directly and can only be obtained through reflection.
It is the underlying implementation of the atomic class

Unsafe CAS operation

Get the offset address of the domain
long offset = unsafe.objectFieldOfSet(xxx.class.getDeclareField(variable name))
Perform CAS operations
unsafe.compareAndSwapInt (modify object, field offset, current value, expected value)
compareAndSwapObject has different methods for different types of data

Shared model memory

ChelseaLiu0822 — Thu, 30 Nov 2023 05:24:50 +0000

Java memory model

JMM is Java Memory Model, which defines the abstract concepts of main memory and working memory. The bottom layer corresponds to the CPU.
Registers, cache, hardware memory, CPU instruction optimization, etc.
JMM is reflected in the following aspects:
Atomic - Guarantees that instructions will not be affected by thread context switching
Visibility - Ensure instructions are not affected by CPU cache
Orderliness - ensures that instructions are not affected by CPU instruction parallel optimization

Visibility

Use volatile to solve visibility problems
Volatile can use Ali to modify member variables or static member variables. It can prevent threads from caching their own work.
To query the value of a variable in the main memory, you must obtain its value from the main memory. When threads operate volatile variables, they directly operate on the main memory.
live
Locking can also solve visibility problems
Modifications of volatile variables by one thread are visible to another thread, and atomicity cannot be guaranteed.
The synchronized statement can not only ensure the atomicity of the code block but also ensure the visibility of the variables within the code block.
The disadvantage is that synchronized is a heavyweight operation with relatively low performance.
volatile is suitable for scenarios where one thread writes and multiple threads read.

Orderliness

The JVM can adjust the execution order of statements without affecting the correctness. This feature is called instruction redundancy.
Arrangement, and instruction rearrangement under multi-threading will affect correctness.

Monitor

ChelseaLiu0822 — Thu, 30 Nov 2023 05:20:01 +0000

Java object header

Klass word points to the class where the object is stored.

Each Java object can be associated with a Monitor object. If synchronized is used to lock the object (weight
level), the pointer to the Monitor object is set in the Mark Word of the object header.

When a thread occupies the lock, the Monitor's Owner will be set to the thread, and there can only be one Owner.
If other threads also execute synchronized, they will enter the EntryList and become Blocked.

lightweight lock

Application scenario: Although an object is accessed by multiple threads, the access times of the multiple threads are staggered (no competition).
contention), then lightweight locks can be used to optimize Lightweight locks are transparent to users, and the syntax is still synchronized

process:

Create a Lock Record object. The stack frame of each thread contains a lock record structure, which can be stored internally.
Mark Word of Lock Object
Let the lock record Object reference point to the lock object, and try to replace the Mark word of Object with cas, so that
The value of Mark word is stored in the lock record
If the cas replacement is successful, the object header stores the lock record address and status 00, indicating that the object is locked by this thread.
If it fails there are two situations:

If other threads already hold the lightweight lock of the Object, it means competition and entry into the expansion process.
If you perform synchronized lock reentry yourself, add another Lock Record as the reentry count

When exiting the synchronized code block (unlocking), the value of the lock record is null, indicating re-entry. At this time, the lock record is reset.
record, indicating that the re-entry count is decremented by one
When exiting the synchronized code block (when unlocking), the value of the lock record is not null. At this time, use cas to change the Mark
The value of Word is restored to the object header
If successful, the unlocking is successful.
Failure, indicating that the lightweight lock has undergone lock expansion or has been upgraded to a weight lock, and the weight lock unlocking process is entered.

expansion lock

If the CAS operation fails when trying to add a lightweight lock, one situation is that other threads are trying to add lightweight locks.
This object has a lightweight lock (with competition). In this case, lock expansion is required to turn the lightweight lock into a heavy lock.

When Thread-1 performs lightweight locking, Thread-0 has already added a lightweight lock to the lock.
At this time, Thread-1 fails to add a lightweight lock and enters the expansion lock process:
Apply for a Monitor lock for the Object object and let the Object point to the weight lock address.
Then let yourself enter the EntryList and Blocked state of the Monitor
When Thread-0 exits the sync block and is unlocked, using cas to restore the Mark word value to the object header will fail. this
will enter the heavyweight lock unlocking process, that is, find the Monitor object according to the Monitor address, and set the Owner to
null, wake up the Blocked thread in EntryList

Spin optimization

When competing for weight locks, you can also use spin for optimization. If the current thread spins successfully (that is, the lock is held at this time)
The thread has exited the synchronization block and released the lock), then the current thread can avoid blocking
It will increase or decrease the number of spins based on whether the last spin was successful, which is more intelligent.

bias lock

When there is no competition for lightweight locks, CAS operations still need to be performed every time the lock is re-entered.
Java 6 introduced biased locking for further optimization. As long as the CAS thread ID is set to the object for the first time,
Mark Word header, and later found that the ID of this thread is itself, which means there is no competition, and there is no need to re-CAS. In the future
As long as no competition occurs, this object is owned by the thread.

When an object is created:

If the bias lock is enabled (enabled by default), then after the object is created, the markword value is 0x05, which is the last 3 digits.

At this time, his thread, epoch, and age are all 0. Bias lock is delayed by default and will not take effect immediately when the program starts. If you want to avoid delay, you can add VM parameters. Number: -XX:BiasedLockinStartupDelay=0 to tighten the delay If the bias lock is not turned on, then after the object is created, the markword value is 0x01 and the last three digits are 001. At this time Its hashcode and age are both 0. The first 54 bits of markword will be assigned when the hashcode is used for the first time. Thread id -XX:-UseBiasedLocking disables biased locking The order of using locks: give priority to bias locks, then light locks, and finally heavy locks. The hashcode() method disables the biased lock because the biased lock does not have enough space to store the hashcode (requires 31 bit) When other threads use the biased lock object, the biased lock will be upgraded to a lightweight lock. The bias lock will also be revoked when wait/notify is called. These two methods are only available for weight locks, so the lock will be upgraded to weight lock

Batch weight bias

If the object is accessed by multiple threads but there is no competition, the object that is biased towards thread T1 will still have a chance to be accessed again.
Bias T2, heavy bias will reset the Thread ID of the object
When the biased lock cancellation threshold exceeds 20 times, the jvm will redirect these objects to the locking thread when accelerating.

Batch undo

When the biased lock threshold is revoked more than 40 times, all objects of the entire class will become non-biasable, and the newly created Objects cannot be biased either

lock elimination

The JIT just-in-time compiler will optimize the bytecode file. When the lock object is a local variable and cannot escape, it will be automatically eliminated.
This lock achieves the purpose of optimization

Wait and notify

If the owner thread condition is not satisfied, call the wait method to enter WaitSet and change to WAITING state.
After the Owner thread calls notify or notifyAll to wake up, the thread enters the EntryList to compete again and will not immediately
get lock wait (long timeout) has a time limit to wait. It will wake up automatically when the time is up and can be woken up in advance by other threads.

The difference between Sleep and Wait

sleep is a Thread method, and wait is an Object method.
Sleep does not need to be used in conjunction with synchronized, but wait needs to be used in conjunction with synchronized.
sleep will not release the object lock while sleeping, but wait will release the object lock while waiting.
The thread status of sleep and wait is the same, both are TIMED WAITING. notify will only randomly wake up the threads in wait, which will cause false wake-ups.

protective pause mode

Application scenario: One thread waits for the execution result of another thread
There is a result that needs to be passed from one thread to another thread, and let them associate a GuardedObject
If there are results constantly flowing from one thread to another, you can use message queue (producer/consumer)
In JDK, the implementation of join and the implementation of Future adopt this mode.
Because it has to wait for the results from the other party, it is classified into synchronous mode.

join principle

Is achieved through protective pause mode

Protected Suspend Mode - Extended

The Futures in the picture are like the mailboxes on the first floor of a residential building (each mailbox has a room number), t0, t2, and t4 on the left are like residents waiting for mail, and t1, t3, and t5 on the right are like postmen.
If you need to use GuardedObject objects between multiple classes, it is not very convenient to pass them as parameters, so design an intermediate class for decoupling. This will not only decouple the [result waiter] and [result producer], but also decouple them at the same time. Supports the management of multiple tasks.

Asynchronous mode-producer/consumer mode

There is no need to produce results and consume results in a thread-to-thread correspondence.
The consumption queue can be used to balance the resources of production and consumption threads
The producer is only responsible for producing results and does not care how the data is processed. The consumer only processes the data.
The message queue has a capacity limit. No more data will be added when it is full, and no more data will be consumed when it is empty.
Various blocking queues in JDK adopt this model.

Park & Unpark

Is a method in the LockSupport class
Compare with Object’s wait & notify
wait, notify and notifyAll must be used together with Object Monitor, but park and unpark are not required.
Park unpark blocks and wakes up in thread units, while notify wakes up a waiting thread randomly.
Park unpark can be unparked first, but wait and notify cannot be notified first.
Each thread has a Parker object, consisting of _counter, _cond and _mutex
_counter is used to determine whether the thread needs to rest. _cond is used to store the resting thread.
When _counter = 0, it means a break is needed. _counter will change to 1 after calling unpark.

Thread status switching:

NEW -> RUNNABLE: t.start() method
RUNNABLE <——> WAITING: Call obj.wait() method t thread from RUNNABLE - -> WAITING

When calling obj.notify(), obj.notifyAll(), t.interrupt()
The competition is successful, t thread goes from WAITING - -> RUNNABLE
Competition failed, t thread went from WAITING - -> BLOCKED
The current thread calls the t.join() method. The current thread starts from RUNNABLE - -> WAITING

RUNNABLE <- ->TIMED_WAITING The thread t ends or the interrupt() method of the current thread is called. The current thread starts from WAITING - ->RUNNABLE When the current thread calls the park method, the current thread will change from RUNNABLE - -> WAITING. Calling unpark (target thread) or the thread's interrupt() method will cause the target thread to start from WAITING - -> RUNNABLE
RUNNABLE <- -> BLOCKED If the competition fails when the t thread acquires the object lock using synchronized (obj), it will start from RUNNABLE - -> BLOCKED After the thread synchronization code block holding the obj lock is executed, all BLOCKED threads on the object will be awakened to compete again. After the competition is successful, the cover thread will change from BLOCKED - -> RUNNABLE and other threads will still be BLOCKED.
RUNNABLE <- -> TERMINATED When all codes of the thread have finished running, enter TERMINATED

multiple locks

Subdivide the lock granularity
Benefits: Can enhance concurrency
Disadvantages: If a thread requires colleagues to obtain multiple locks, deadlocks may easily occur.

deadlock

Thread t1 acquires the A object lock, and then wants to acquire the B object lock.
Thread t2 acquires the B object lock, and then wants to acquire the A object lock.
To monitor deadlocks, you can use the jconsole tool, or use jps to locate the process ID, and then use jstack to locate the deadlock.

livelock

The two threads change the end conditions of the object to each other, and neither one can end it in the end.

Hunger problem:

Because the priority of a thread is too low, it is never scheduled to be executed by the CPU and cannot be terminated.
Sequential locking can solve the deadlock problem, but it will cause starvation problems, which can be solved with ReentrantLock.

ReentrantLock

For synchronized, it has the following characteristics:
Interruptible
Timeout can be set
Can be set to fair lock
Support multiple condition variables
Like synchronized, both support reentry.
To create a ReentrantLock object first

Can be interrupted:
While t1 is waiting, other threads can use the interrupt() method to interrupt the waiting of the t1 thread.
The lock() method is uninterruptible, and the lockInteruptibly() method can be interrupted.

lock timeout

If you still cannot obtain the lock after waiting for a period of time, give up waiting.
The tryLock() method returns a boolean value, false indicates that the lock is not acquired, and it also supports interrupting

fair lock

ReentrantLock is unfair by default (released locks are not allocated in the order of the blocking queue). The original intention is that
In order to solve the starvation problem, pass true during construction to become a fair lock. Fair lock will reduce concurrency.

condition variable

When the conditions are not met, enter waitSet to wait.
ReentrantLock supports multiple condition variables. Different conditions have different waitSets (lounges). You can follow
waitSet(lounge) to wake up
Create a condition variable:
Condition cond1 = lock.newCondition()
Need to obtain the lock before await
cond1.await() enters waiting
cond1.singal() wakes up a thread in cond1
cond1.singalAll() wakes up all threads in cond1

shared model

ChelseaLiu0822 — Thu, 30 Nov 2023 04:39:51 +0000

critical section

Multiple threads access shared resources, and when instructions are interleaved during read and write operations on shared resources, concurrent problems may occur.
question
If there are multi-threaded read and write operations on shared resources within a code block, this code is called a critical section.

race condition

Multiple threads are executing in the critical section. Due to the different execution sequences of the code, it is impossible to predict, which is called a race.
state conditions

Methods to avoid race conditions in critical sections:

Blocking solutions: synchronized, lock
Non-blocking solution: atomic variables

synchronized

The object in synchronized can be imagined as a room with only one entrance, and only one person can enter at a time.
Make calculations
When the t1 thread owns the lock and enters the room for calculation, and the context switch occurs, when t2 wants to enter the room, it finds that there is no thread.
If you have a key and cannot enter, it will become blocked (blocked state), and t1 will continue to perform calculations, and only t1 will be released.
After locking, t2 can get the lock and enter the room for calculation.
synchronized actually uses object locks to ensure the atomicity of the critical section code. The code in the critical section is not exposed to the outside world.
Divisible and will not be interrupted by thread switching
Synchronized on member methods lock this object
Synchronized on static methods locks class objects
Member variables and static variables need to consider thread safety when they are shared and have read and write operations.
If the reference object of a local variable escapes the scope of the method, thread safety needs to be considered.

Thread safety class

String
Integer all wrapper classes
StringBuffer
Random
Vector
Hashtable
Classes under the java.until.concurrent package
When multiple threads call a method of their same instance, it is thread-safe, and each of their methods is Atomic, but their combination method is not necessarily atomic.

String and Integer are both immutable classes because their internal state cannot be changed, so their methods are Thread safe

The state of the thread

ChelseaLiu0822 — Thu, 30 Nov 2023 04:35:26 +0000

from OS

When an application is to be processed, then it creates a thread.
It is then allocated the required resources(such as a network) and it comes in the READY queue.
When the thread scheduler (like a process scheduler) assign the thread with processor, it comes in RUNNING queue.
When the process needs some other event to be triggered, which is outsides it’s control (like another process to be completed), it transitions from RUNNING to WAITING queue.
When the application has the capability to delay the processing of the thread, it when needed can delay the thread and put it to sleep for a specific amount of time. The thread then transitions from RUNNING to DELAYED queue.
An example of delaying of thread is snoozing of an alarm. After it rings for the first time and is not switched off by the user, it rings again after a specific amount of time. During that time, the thread is put to sleep.

When thread generates an I/O request and cannot move further till it’s done, it transitions from RUNNING to BLOCKED queue.
After the process is completed, the thread transitions from RUNNING to FINISHED.
The difference between the WAITING and BLOCKED transition is that in WAITING the thread waits for the signal from another thread or waits for another process to be completed, meaning the burst time is specific. While, in BLOCKED state, there is no specified time (it depends on the user when to give an input).
https://www.geeksforgeeks.org/thread-states-in-operating-systems/

from Java

NEW – a newly created thread that has not yet started the execution
RUNNABLE – either running or ready for execution but it’s waiting for resource allocation
BLOCKED – waiting to acquire a monitor lock to enter or re-enter a synchronized block/method
WAITING – waiting for some other thread to perform a particular action without any time limit
TIMED_WAITING – waiting for some other thread to perform a specific action for a specified period
TERMINATED – has completed its execution

Main thread and daemon thread

ChelseaLiu0822 — Thu, 30 Nov 2023 04:28:02 +0000

As long as there are threads running, the Java process will not end.
For daemon threads, as long as other non-daemon threads have finished running, even if the code of the daemon thread has not finished executing,
It will force the end of setDaemon(true) and set it as a daemon thread.
The garbage collector is a daemon thread
The Acceptor and Poller threads in Tomcat are both daemon threads

interrupt method

ChelseaLiu0822 — Thu, 30 Nov 2023 04:27:22 +0000

Interrupting a blocked thread (wait, sleep, join) will clear the interruption status
Interrupting a normally running thread will not clear the interruption status.
The interrupt state can be used to stop a thread.

Two-stage termination model

Gracefully terminate the T2 thread in the T1 thread and give T2 a chance to take care of the aftermath (release locks and resources)

Interrupting the park thread will not clear the interruption status
After the interrupt flag is true, calling park again will fail.

Not recommended method

Will destroy the synchronized code block and cause deadlock