Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cuda heat example w quaditer #913

Draft
wants to merge 15 commits into
base: master
Choose a base branch
from

Conversation

Abdelrahman912
Copy link
Contributor

Heat Example Prototype using CUDA.jl and StaticCellValues

end

Kgpu = CUDA.zeros(dh.ndofs.x,dh.ndofs.x)
gpu_dh = GPUDofHandler(dh)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe that's the plan anyway but wouldn't it be nicer to write Adapt rules for DofHandler and Grid which return adapt_structure for the GPU structs?

function Adapt.adapt_structure(to,dh:DofHandler)
    return adapt_structure(GPUDofHandler(cu(Int32.(dh.cell_dofs)),GPUGrid(dh.grid)))
end

and then just use the "normal" structs from a user perspective that are automatically converted when needed

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am still a bit split about this, because it contains a performance pitfalls. If we have repeated assembly, then the dof handler will be converted and copied to GPU for each assembly instead of only once.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Am I missing something or wouldn't the adapt call not also happen for each assembly kernel launch for the GPUDofHandler in that case?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should not happen, because we do not need to adapt the GPUDofHandler (it is already a GPU datastructure).

end


gm = static_cellvalues.gm
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe this is still a work in progress but the mix of functions and global variables make it kind of confusing to read the code.

@Abdelrahman912
Copy link
Contributor Author

What I did for now, and it's still work in progress:

  1. I added some higher-level abstractions and some restructuring to match the original example (still need some refactoring).
  2. I used the QuadratureValuesIterator and edited the StaticCellValue object to be compatible with the GPU.

This still work in progress and as my discussion with @termi-official last week I still need to work on the assembler, coloring algorthim.

Some problems I have encountered that might be so straightforward to tackle:

  1. Grid object contains Dict type which is not GPU compatible.

@termi-official
Copy link
Member

Great to see some quick progress here!

Some problems I have encountered that might be so straightforward to tackle:

1. `Grid` object contains `Dict` type which is not GPU compatible.

I think that is straight forward to solve. We never really need the Dicts directly during assembly. We should be able to get away by just convert the Vectors (once) to GPUVectors and run the assembly with these. This might require 2 structs. One holding the full information (e.g. GPUGrid) and one which we use in the kernels (e.g. GPUGridView). Maybe the latter could be something like

struct GPUGridView{TEA, TNA, TSA <: Union{Nothing, <:AbstractVector{Int}, <: AbstractVector{FaceIndex}, ..., TCA} <: AbstractGrid (?)
    cells::TEA
    nodes::TNA
    subdomain::TSA
    color::TCA
end

where subdomain just holds the data which we want to iterate over (or nothing for all cells) and color is a vector for elements with one color of the current subdomain.

@KnutAM
Copy link
Member

KnutAM commented May 23, 2024

A longer-term thing just to throw out the idea, but perhaps a more slim Grid could be nice?

struct Grid{dim, C, T, CV, NV, S}
    cells::CV
    nodes::NV
    gridsets::S
    function Grid(cells::AbstractVector{C}, nodes::AbstractVector{Node{dim, T}}, gridsets) where {C, dim, T}
        return new{dim, C, T, typeof(cells), typeof(nodes), typeof(sets)}(cells, nodes, gridsets)
     end
end
struct GridSets
    facetsets::Dict{String, OrderedSet{FacetIndex}}
    cellsets::Dict{String, OrderedSets{Int}}
    ....
end

allowing also gridsets=nothing

@termi-official
Copy link
Member

termi-official commented Jun 4, 2024

A longer-term thing just to throw out the idea, but perhaps a more slim Grid could be nice?

struct Grid{dim, C, T, CV, NV, S}
    cells::CV
    nodes::NV
    gridsets::S
    function Grid(cells::AbstractVector{C}, nodes::AbstractVector{Node{dim, T}}, gridsets) where {C, dim, T}
        return new{dim, C, T, typeof(cells), typeof(nodes), typeof(sets)}(cells, nodes, gridsets)
     end
end
struct GridSets
    facetsets::Dict{String, OrderedSet{FacetIndex}}
    cellsets::Dict{String, OrderedSets{Int}}
    ....
end

allowing also gridsets=nothing

I thought of this quite a bit already, whether we should have our grid in the form

struct Grid{dim, C, T, CV, NV, S, TT}
    cells::CV
    nodes::NV
    subdomain_info::S
    function Grid(cells::AbstractVector{C}, nodes::AbstractVector{Node{dim, T}}, subdomain_info) where {C, dim, T}
        return new{dim, C, T, typeof(cells), typeof(nodes), typeof(sets)}(cells, nodes, subdomain_info)
     end
end

where subdomain info contains any kind of subdomain information. This could also include potentially some optional topology information which we need for some problems. In the simplest case it would be just facesets and cellsets.

However, we should do this in a separate PR. What do you think @fredrikekre ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants